Method and apparatus for multiple raw sensor image enhancement through georegistration

ABSTRACT

A method and apparatus for generating an image from raw sensor data. In one embodiment, the method comprises reading a plurality of raw sensor data sets from one or more sensors at a plurality of sensor inertial states, generating an estimate of each of the plurality of sensor inertial states, and while retaining each of the raw sensor data sets, generating an image, the image generated at least in part from the plurality of estimated sensor inertial states and the plurality of raw sensor data sets, and generating an updated estimate of at least one of the sensor inertial states, the updated estimate of the at least one of the sensor inertial states generated at least in part from the generated image and the plurality of estimated sensor inertial states. Finally, an enhanced image is generated from the retained raw sensor data sets and the updated estimate of the at least one of the sensor inertial states.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following co-pending and commonlyassigned patent application, which applications are incorporated byreference herein:

application Ser. No. ______, entitled “METHOD AND APPARATUS FOR RAWSENSOR IMAGE ENHANCEMENT THROUGH GEOREGISTRATION,” filed on same dateherewith, by Daniel T. Phillips.

BACKGROUND 1. Field

The present disclosure relates to systems and methods for generatingimages from raw sensor data, and in particular to a system and methodfor enhancing such image generation through georegistration.

2. Description of the Related Art

Image registration is the process of transforming different sets ofimage data into a single coordinate system. The image data used in theregistration process typically comprises multiple images, ofsubstantially the same scene, and such images may be taken at the sametime, or at different times or viewpoints. Data registration techniquesare used in computer vision, medical imaging, military automatic targetrecognition, and compiling and analyzing images and data fromsatellites. Registration is necessary in order to be able to compare orintegrate the data obtained from these different measurements.

In some cases, images may be registered in conjunction with referenceviews synthetic models of the same scene depicted in the image(s). Forexample, U.S. Patent Publication 2005/0220363, which is herebyincorporated by reference herein, depicts a processing architecture forautomatically registering images with a reference image database and/ora digital elevation model (DEM).

The collection of sensor data and subsequent image processing isdistorted by the uncertainty of the location of the sensor taking theraw sensor measurements. This is due to uncertainties in the navigationsystem of the platform upon which the sensor is mounted, errors in theimage registration process, and that only a limited set of referenceelevation data is typically available to process the image.

What is needed is a system and method that reduces such distortion. Sucha system and method is disclosed below.

SUMMARY

To address the requirements described above, this document discloses asystem and method for generating an image from raw sensor data. In oneembodiment, the method comprises reading a plurality of raw sensor datasets, each raw sensor data set read from one or more sensors at aplurality of sensor inertial states, generating an estimate of each ofthe plurality of sensor inertial states, and retaining each of the rawsensor data sets. Each of the raw sensor data sets are retained whilegenerating an image at least in part from the plurality of estimatedsensor inertial states and the plurality of raw sensor data sets andgenerating an updated estimate of at least one of the sensor inertialstates, wherein the updated estimate of the at least one of the sensorinertial states is generated at least in part from the generated imageand the plurality of estimated sensor inertial states. Finally anenhanced image is generated from the retained raw sensor data sets andthe updated estimate of the at least one of the sensor inertial states.Another embodiment is evidenced by means for performing the aboveoperations.

Still another embodiment is evidenced by an apparatus for generating animage from raw sensor data that comprises a sensor for generating rawdata, an image processor, communicatively coupled to the sensor, forreading a plurality of raw sensor data sets from the sensor, each rawsensor data set read from the sensor at a plurality of sensor inertialstates, an inertial navigation system, communicatively coupled to thesensor, for generating an estimate of each of the plurality of sensorinertial states, an image processor, communicatively coupled to thesensor and the inertial navigation system, for generating an image, theimage generated at least in part from the plurality of estimated sensorinertial states and the plurality of raw sensor data sets; and ageoregistration system, communicatively coupled to the inertialnavigation system and the image processor, for generating an updatedestimate of at least one of the sensor inertial states, the updatedestimate of the at least one of the sensor inertial states generated atleast in part from the generated image and the plurality of estimatedsensor inertial states. In this embodiment, the image processorgenerates an enhanced image from the retained raw sensor data sets andthe updated estimate of the at least one of the sensor inertial statesand the sensor retains each of the raw sensor data sets while the imageis generated from the raw sensor data and the updated estimate of the atleast one of the sensor inertial states is generated. Still anotherembodiment is evidenced by an apparatus having a processor and acommunicatively coupled memory storing processor instructions forperforming the foregoing operations.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments of the present inventionor may be combined in yet other embodiments, further details of whichcan be seen with reference to the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is a diagram illustrating the operation of a sensor platform;

FIG. 2 is a diagram presenting an exemplary georegistration system;

FIG. 3 is a diagram illustrating a more detailed embodiment of ageoregistration system

FIGS. 4A-4D are a diagrams presenting exemplary process steps that canbe used to generate an image from raw sensor data;

FIG. 5 is a diagram of one embodiment of an improved georegistrationsystem; and

FIG. 6 illustrates an exemplary computer or system 600 that could beused to implement processing elements.

DESCRIPTION

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and which is shown, by way ofillustration, several embodiments. It is understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the present disclosure.

Overview

Unlike existing systems and methods for generating images from rawsensor data, the system and method described herein retains raw sensorimage data (e.g. Synthetic Aperture Radar (SAR) IQ Collection Data).Upon successful georegistration to high resolution reference imagery,the raw sensor data is reprocessed with calculated enhanced sensorlocation and reference imagery data resulting in an enhanced outputsensor image of greater accuracy and reduced distortions, for example,reprocess raw sensor SAR data using an algorithm like back projectionwhose main benefit would be seen in high-resolution collections withsignificant of out-of-plane acceleration. Other distortions that couldbe improved include foreshortening and layover distortions, which areproblematic in SAR applications.

Platform

FIG. 1 is a diagram illustrating the operation of a sensor platform 104(alternatively referred to hereinafter as platform). The sensor platform104 may operate in the Earth's atmosphere or space, and may be unmannedor manned. In one embodiment, the sensor platform is an unmanned airvehicle (UAV). The platform 104 typically includes a sensor 106 (whichmay be mounted on a rotatable turret 105) having a field of view (FOY)108 that surveils targets 110 on or near the ground.

The platform 104 also typically comprises a processor 112communicatively coupled to a memory 116 storing processor instructionsfor performing the operations described herein, as well as a means fortaking inertial measurements of the motion of the platform 104, andtranslating these measurements into an estimate of the inertial state ofthe platform 104 or, via suitable coordinate transformations, the sensor106 such as an inertial navigation system (INS) 114. Typically, the INS114 comprises an inertial reference unit (IRU) that includes threeaccelerometers providing acceleration measurement in three orthogonaldirections, and three rate sensors such as gyros that sense rotation inthree orthogonal directions. Typically, the INS 114 may also compriseanother receiver for receiving global navigation information, such as isavailable from systems such as the global positioning system (GPS). GPSincludes a plurality of satellites 102A-102N which transmit signals thatthe INS 114 uses to assist in the determination of its position ininertial space. In some applications, the INS 114 may also comprise astar tracker or other means for estimating position.

In one embodiment, the platform 104 also comprises a receiver 117,communicatively coupled to the processor 112, for transceiving commandsand other information with a ground station 118. Using computers andother equipment in the ground station 118, users 120 provide commands tothe platform 104 and receive data, including data obtained by the sensor106.

Sensor Embodiments

In one embodiment, the sensor 106 comprises a SAR which scans a scene109 including the target 110, and constructs an image of the target 110and/or scene 109 with raw sensor data generated from such scans.

SAR is a coherent mostly airborne or space borne side looking radarsystem which utilizes the flight path of the platform 104 to simulate anextremely large antenna or aperture electronically, and that generateshigh-resolution remote sensing imagery. Over time, individualtransmit/receive cycles are completed with the data from each cyclebeing stored electronically. The signal processing uses magnitude andphase of the received signals over successive pulses from elements of asynthetic aperture. After a given number of cycles, the stored data isrecombined (taking into account the Doppler effects inherent in thedifferent transmitter to target geometry in each succeeding cycle) tocreate a high resolution image of the terrain being over flown.

SAR works much like a phased array, but instead of a large number of theparallel antenna elements of a phased array, SAR uses one antenna (whichproduces a beam that can steered mechanically or electronically) andtime-multiplexes the measurements to simulate a large aperture sensor.The different geometric positions of the antenna elements are result ofthe moving platform 104.

The image processor stores all the radar returned signals, as amplitudesand phases, over a time period, with each signal representing data takenfrom an associated position in inertial space. Now it is possible toreconstruct the signal which would have been obtained by an antenna oflength v·T, where v is the platform speed, and T is the time period. Asthe line of sight direction changes along the radar platform trajectory,a synthetic aperture is produced by signal processing that has theeffect of lengthening the antenna. Making T large makes the “syntheticaperture” large and hence a higher resolution can be achieved.

As the target 110 first enters the radar beam, the backscattered echoesfrom each transmitted pulse begin to be recorded. As the platformcontinues to move forward, all echoes from the target for each pulse arerecorded during the entire time that the target is within the beam. Thepoint at which the target leaves the view of the radar beam some timelater, determines the length of the simulated or synthesized antenna.The synthesized expanding beamwidth, combined with the increased time atarget is within the beam as ground range increases, balance each other,such that the resolution remains constant across the entire swath. Theachievable azimuth resolution of a SAR is approximately equal toone-half the length of the actual (real) antenna and does not depend onplatform altitude (distance). The process of generating the image may bethought of as a process wherein the incoming raw sensor data is providedto a shift register with the arrival of each new raw sensor data causingthe previously measured raw data to be shifted to an adjacent shiftregister element, and the new data placed into the vacated element. Thedata from each shift register is then combined using an arithmeticoperation or processing function, with the resolution of the imageimproving as each new raw image data set is processed (a resultexpected, as each new measurement increases the “aperture” associatedwith the total of the image data. Typically, once the image isgenerated, the data in the shift registers is deleted or written over,thus eliminating the possibility that improvements in estimates of theinertial state of the sensor (generated from the image viageoregistration) can be used to improve the quality of the image.

Hence, SAR generates an image using successive instances of raw imagedata, which combined using SAR image processing techniques, combineseach data instance, taken at different points in inertial space tocreate a complete image. The accuracy of the resulting image dependsupon several factors, including the accuracy of any determination of theplatform 104 position when each successive image is taken. Therequirements for such a system is a stable, fully coherent transmitter,an efficient and powerful SAR image processor, and knowledge of theflight path and/or velocity of the sensor.

SAR is subject to slant-range distortions, including foreshortening,layover, and shadowing effects. Such slant-range distortion occursbecause the radar is measuring the distance to features in slant-rangerather than the true horizontal distance along the ground. This resultsin a varying image scale, moving from near to far range. Foreshorteningoccurs when the radar beam reaches the base of a tall feature tiltedtowards the radar (e.g. a mountain) before it reaches the top. Becausethe radar measures distance in slant-range and not horizontal range, theslope appears compressed and the length of the slope will be representedincorrectly at the image plane. Layover occurs when the radar beamreaches the top of a tall feature before it reaches the base. The returnsignal from the top of the feature will be received before the signalfrom the bottom. As a result, the top of the feature is displacedtowards the radar from its true position on the ground, and “lays over”the base of the feature. The shadowing effect increases with greaterincident angle 0, just as our shadows lengthen as the sun sets.Compensating for all of such slant range distortions is possible, butsuch compensation depends at least in part upon an accurate knowledge ofthe inertial state of the sensor (e.g. its position and/or velocity). Adescription of the basic principles of SAR is presented athttp://www.radartutorial.eu/20.airborne/ab07.en.html

In another embodiment, the sensor 106 comprises a planar imaging sensorwhich also scans the scene 109 including the target 110, and constructsan image of the target 110 and/or scene 109 with raw sensor datagenerated from such scans. Typically, the planar imaging sensorcomprises a camera permitting the capture of a sequence of images, suchas a movie camera, and may sense energy in visible, infrared (IR) orultraviolet wavelengths. In some applications the planar sensor collectsdata in other wavelengths across the electromagnetic spectrum.Typically, the imaging sensor 106 can be oriented by maneuvering theplatform 104 in pitch, yaw and roll, and may also be orientedindependent of the platform 104 body in tilt and pan directions. Suchtilting and panning may be accomplished electronically or mechanically,using the turret 105 or similar structure.

Georegistration System

FIG. 2 is a diagram presenting an exemplary georegistration system 200.The georegistration system comprises the sensor 106, communicativelycoupled to an image processor 204. The image processor 204 processes theraw sensor data 202 to generate the image 206. Typically, the imageprocessor 204 is a special purpose processor dedicated to the purpose ofgenerating the image 206, and is distinct from platform processor 112.However, the platform processor 112 may also be used to perform thefunctions of the image processor 204.

The georegistration system 200 also comprises the INS 114,communicatively coupled to the image processor 204. As described above,the INS 114 generates estimates of the inertial state of the platform104, and by suitable coordinate transformation, the sensor 106, overtime. The sensor inertial state 208 may include, for example, theposition, velocity, acceleration, attitude or attitude rate of thesensor 106. These states may be expressed with respect to an inertialcoordinate space, which may be in Cartesian, polar, or other coordinateschemes (NED, ECEF, System/Sensor Body). For simplicity, one or moreestimates of the position, velocity, acceleration, attitude, or attituderate of the sensor corresponding to one or more raw sensor datacollection events is defined as Sensor Inertial State and Raw SensorData, respectively.

The image processor 204 receives the raw sensor data 202 and processesthis data to generate an image. In the embodiment wherein the sensor 106is a SAR, the image processor 204 receives raw IQ sensor data 202 fromthe sensor 106 taken at a plurality of physical sensor 106 locations,and using an estimate of the physical location of the sensor 106 wheneach such raw sensor data was taken, generates image 206.

The image 206 and sensor inertial state 208 is provided to ageoregistration module 210. The georegistration module 210 generates ageoregistered sensor image 212 from the sensor inertial state 208, theimage 206, and reference imagery and/or other reference geospatial data(e.g. Elevation) from a reference image database.

FIG. 3 is a diagram illustrating a more detailed embodiment of ageoregistration system 300. Platform parameters 322 (which can betransformed to the sensor inertial state 208) and sensor parameters 332(e.g. field of view or FOV) are provided to a sensor footprint analysismodule 324 to generate an estimate of the location of the scene depictedby the image and the sensor footprint. That information is provided toan area of interest (AOI) extraction module 326, which uses thereference image database 214 and a reference digital elevation model(DEM) database 340 to generate a reference chip 330 and a DEM chip 342.

The platform parameters 322 and sensor parameters 320 are also providedto a sensor perspective analysis module 350, which generates perspectiveparameters 352 that are used, along with a sensor transform model 354describing the relationship between the platform and the sensor, togenerate a transform for transforming reference images to the sensorperspective. The reference chip 330 and DEM chip 342 are provided to anorthoimage construction module 344 to generate a reference orthoimagechip 348 and a reference DEM chip 346, which are transformed into thesensor perspective by sensor perspective module 356 to generate aperspective reference image 358. The perspective reference image 358 ismatched with a sensor image generated from the raw sensor data by imagematching module 360 to generate match parameters. These match parametersdefine a matching function 362 which reflects a transform (e.g.translation, rotation, inverse perspective, etc.) to match the sensorimage with the perspective reference image. An exemplary embodiment ofthe foregoing system is further described in U.S. Patent Publication2005/0220363, which is hereby incorporated by reference.

This georegistered image 212 can be used to increase the accuracy of thedetermination of the inertial state of the sensor 106 occurring at thetime(s) of collection and hence the platform 104 itself. Importantly,and as discussed below, if the raw sensor data 202 is retained, thismore accurate data regarding the inertial state of the sensor 106 canalso be used to generate an enhanced (e.g. better quality) image 206. Toaccomplish this, the raw sensor data 202 must be retained or otherwisestored so the image processing used to generate the image 206 isrepeated (e.g. performed again) using the more accurate sensor 106inertial state data. Hence, upon successful georegistration to highresolution reference imagery, the raw sensor data 202 is reprocessedwith calculated enhanced sensor location and reference imagery dataresulting in an enhanced output sensor image of greater accuracy andreduced distortions (e.g. out of plane acceleration, foreshortening andlayover).

Enhanced Image Generation

FIGS. 4A-4D are a diagrams presenting exemplary process steps that canbe used to generate an image 206 from raw sensor data 202. Turning firstto FIG. 4A, in block 402, the raw sensor data 202 is read from thesensor 106. In block 404, a estimate of the sensor inertial state 208 isgenerated by the INS 114. Typically, the process of reading the rawsensor data 202 and the process of generating an estimate of theinertial state of the sensor 106 are performed concurrently byindependent processes, with digital data from each process time-taggedso as to allow the raw sensor data 202 to be correlated with theestimated sensor inertial state 208 when the raw sensor data 202 wasread. Hence, the inertial estimate sensor inertial state 208 may begenerated during or concurrently with the reading of the raw sensor datafrom the sensor 106.

As depicted in block 406, the raw sensor data 202 is retained orotherwise stored for later use in generating an updated (or enhanced)version of image 206. In one embodiment, this is accomplished by storingthe raw sensor data 202 in a memory (e.g. memory 502 illustrated in FIG.5) separate from the sensor 106. In other embodiments, this isaccomplished by storing the raw sensor data 202 in a memory of thesensor 106 itself, in a memory that is a part of or is accessible to theimage processor 204, or the platform processor 112.

As shown in block 408, an image 206 is generated at least in part fromthe raw sensor data 202. This can be accomplished, for example, usingthe image processor 204 illustrated in FIG. 5. In one embodiment, theraw sensor data comprises data obtained from a planar imaging sensorsuch as an infra-red or visible light sensor having an array of elementssensitive to infra-red or visible light. In this embodiment, the imagecan be generated from the raw sensor data without the estimated inertialsensor state, as the location of the sensor when the raw data was readis not required to generate the image. While not required to generatethe image, this data may be utilized as a second sensor data input toenhance the estimate of the inertial state of the platform as in FIG.4C.

In another embodiment, the estimated sensor inertial state 208 isrequired for the image processor 204 to generate the image. For example,in applications where the image 206 is generated by combining raw sensordata 202 from taken from a plurality of sensor 106 locations (e.g.synthetic aperture systems such as SAR), the image 206 is generatedusing not only the raw sensor data 202, but also the inertial state ofthe sensor 106 when the raw sensor data 202 was taken.

As shown in block 410, an updated estimate of the sensor inertial stateat least in part from the generated image 206 and the estimated sensorinertial state. In one embodiment, the updated estimate of the sensorinertial state is generated by georegistering the image generated inblock 408, and using the georegistered image to obtain the updateestimate of the sensor inertial state. This further described below withreference to FIG. 5. Finally, in block 412, an enhanced image isgenerated from the retained raw sensor data and the updated estimate ofthe sensor inertial state.

In one embodiment, the enhanced image generated in block 412 is providedto block 410 and used to further improve the updated estimate of thesensor inertial state. Since the raw sensor data is still available,this allows for the generation of a further enhanced image when theoperations of block 412 are repeated. This process can be iterativelyrepeated, improving the generated image and the and the estimate of thesensor inertial state until such iterations no longer sufficientlyimprove the estimate of the sensor inertial state.

The foregoing process may be repeated for additional raw sensor data andadditional estimates of the sensor inertial state. This may be applied,for example, in embodiments using synthetic aperture techniques such asSAR (where multiple raw sensor data is taken from different locations atdifferent points in time) or multiple sensor embodiments (where multipleraw sensor data is taken from the different locations at the same ornearly the same point in time).

FIG. 4B is a diagram illustrating the use of additional raw sensor dataand estimates of the sensor inertial state. This embodiment is useful,for example, in selected SAR applications. In block 422, second raw datais read from the sensor. In this embodiment, the sensor is the samesensor that read the first raw data described above (but at a differentlocation at a different time). In block 424, an estimate of a secondsensor inertial state is generated. The second sensor inertial state isthe state of the same sensor, but at a different location (and time)than the sensor inertial state described above. In block 426, the secondraw sensor date is retained (the first raw sensor data has also beenretained for use).

In block 428 an image is generated. This image is generated at least inpart from the updated estimate of the sensor inertial state, theestimate of the second sensor inertial state, the raw sensor data andthe second raw sensor data. For example, in the SAR embodiment, thefurther generated image is generated from the first raw data taken atthe first sensor location and the second raw data taken at the secondlocation, as well as the state of the sensor at the locations where theraw data was taken.

In block 430, an updated estimate of the second inertial sensor state isgenerated. This updated estimate of the second inertial sensor state isgenerated at least in part from the further generated image generated inblock 428 and the estimated second sensor inertial state.

Finally, in block 432, a further enhanced image is generated, using theretained raw sensor data, the retained second raw sensor data, theupdated estimate of the sensor inertial state and the updated estimateof the second sensor inertial state.

Similarly to the process shown in FIG. 4A, the foregoing process may berepeated for additional raw sensor data and additional estimates of thesensor inertial state. This may be applied, for example, in embodimentsusing synthetic aperture techniques such as SAR (where multiple rawsensor data is taken from different locations at different points intime) or multiple sensor embodiments (where multiple raw sensor data istaken from the different locations at the same or nearly the same pointin time).

FIG. 4C is a diagram illustrating the use of additional raw sensor dataand estimates of the inertial state in an embodiment using two sensors.In this embodiment, the first raw sensor data and the second raw sensorare taken from two different sensors, and may be taken at the same ordifferent times. Block 432 reads second raw sensor data from a secondsensor, and block 434 generates an estimate of the second sensorinertial state. Block 436 retains the second raw sensor data at least aslong as the operations of blocks 438 and 440 are performed (the firstraw sensor data has also been retained for use). In block 438, the imageis further generated at least in part from the updated estimate of thesensor inertial state (of the first sensor), the second estimatedinertial state (of the second sensor), and the raw data from the firstsensor and the second sensor. In block 440, an updated estimate of thesecond inertial sensor state (of the second sensor) is generated atleast in part from the further generated image and the second sensorinertial state (of the second sensor). Finally, in block 442, a furtherenhanced image is generated from the first retained raw sensor data, thesecond retained raw sensor data.

Again, similar to the embodiments discussed in connection with FIGS. 4Aand 4B, the enhanced image generated in block 442 may be provided toblock 440 and used to further improve the updated estimate of the secondsensor inertial state. Since the second raw sensor data is stillavailable, this allows for the generation of a further enhanced imagewhen the operations of block 442 are repeated.

FIG. 4D is diagram of another embodiment of exemplary process steps thatcan be used to generate an image 206 from raw sensor data 202. In thisembodiment, an image of a target 110 covering a certain area 109 at acertain resolution is desired. A prediction is generated regarding howlong it will take to collect the image (time) and verifying that theimage is predicted to remain within the field of view 108 of the sensor106 during the collection. Greater resolution is achieved when more datapoints at different sensor locations are collected, which typicallyreflects in a greater period of time to collect the data. As the sensorplatform 104 moves, the sensor 106 begins collection collecting raw data(in the case of a SAR, each set of raw data is obtained from aradio-frequency pulse) at a plurality of sensor positions or inertialstates. The INS 114 generates estimates of the sensor position orinertial state for each set of raw data corresponding to a pulse. Eachset of raw sensor data 202 is then associated with the estimate of theinertial position of the sensor 106. This may be accomplished, forexample, by correlating the recorded time that the raw sensor data 202was collected from each pulse and the time of the estimate of theinertial state of the sensor 106.

In block 458, an image 206 is generated from at least in part from theraw sensor data 202 (which includes the data sets from each of themultiple pulses) and the estimate of the inertial states of the sensor208 corresponding to each of the raw sensor data 202 sets. In block 460,an updated estimate of the sensor inertial states is generated at leastin part from the generated image and the estimated sensor inertialstates. In one embodiment, this is accomplished by generating ageoregistered image 212 from the generated image 206 and the referenceimage database 214, then generating the updated estimate of the sensorlocation(s), using the georegistered image 212. In block 462, anenhanced image is generated from the retained raw sensor data sets 456and the updated estimate of the sensor inertial states.

Again, similar to the embodiments discussed in connection with FIGS.4A-4C, the enhanced image generated in block 462 may be provided toblock 460 and used to further improve the updated estimate of the sensorinertial states. Since the raw sensor data is still available, thisallows for the generation of a further enhanced image when theoperations of block 462 are repeated.

Further, the improved estimate of the sensor inertial state can be usedto improve the estimated inertial state for future raw sensor data 202set collections, thus further improving the generated image 206.

In the embodiments discussed in reference to FIGS. 4A-4D, the estimatedsensor inertial state may include different data, depending on thesensor 106 used. For example, in embodiments in which the raw sensordata is taken at different sensor locations by the same sensor, (forexample, SAR), the estimated sensor inertial state includes the sensorlocation in inertial space, and may also include the sensor velocity ininertial space. In embodiments (for example, using a planar sensorhaving an array of pixels sensitive to energy in IR, uV, or visiblelight bandwidths), the sensor inertial state may include the sensorlocation and the sensor attitude when the raw sensor data is read.

FIG. 5 is a diagram of one embodiment of an improved georegistrationsystem 500. Blocks 410, 430, 440 and 460 of FIGS. 4A, 4B, 4C, and 4Drespectively, describe operations in which an updated estimate of thesensor inertial state (or the second inertial sensor state) is generatedat least in part from the related image. This may be accomplished by thegeoregistration module 210, which generates a georegistered image fromthe generated image and a reference image database 214 and theaforementioned reference DEM database referred to above. The updatedsensor inertial state is then generated at least in part from thegeoregistered image. This can be accomplished by comparing thegeoregistered image to the known features in the DEM database 340 andgeoregistered image as described in U.S. Patent Publication 2005/0220363to provide additional information regarding the location of the sensorwhen the raw sensor data used to generate the image was read. Suchcomparison may be performed by a hardware, software, or firmware module506 of the INS 114, a hardware, software, or firmware module 504 of theimage processor 204 and/or a hardware, software, or firmware module ofthe platform processor 112.

Finally, since an enhanced image is available from the process describedabove, an enhanced georegistered image may be generated from theenhanced image and the updated sensor location(s). This is accomplishedby providing the enhanced image 206 (created using the updated sensorinertial state 208) and the updated sensor inertial state 208 itself tothe georegistration module 210. Since the georegistration module 210 nowhas an enhanced image and an updated (and more accurate) sensor inertialstate 208, an enhanced georegistered sensor image 212 can be generated.Of course, this enhanced georegistered sensor image 212 can be used tofurther increase the accuracy of the sensor inertial state 208 which canin turn be used to improve the image 206, which then allows generationof a further enhanced georegistered sensor image 212. Hence, byretaining the raw sensor data 202 so that it is available to bereprocessed using sensor inertial state 208 data with improved accuracymade possible by the georegistration process, a closed loop ofprocessing is defined which continually improves both the image and theestimate of the sensor states.

Processing Environment

FIG. 6 illustrates an exemplary computer or system 600 that could beused to implement processing elements of the above disclosure, includingthe platform processor 112, image processor 204, georegistration module210, portions of the INS 114, receiver 117 and ground station 118. Thecomputer 602 comprises a processor 604 and a memory, such as randomaccess memory (RAM) 606. In embodiments requiring a human interface, thecomputer 602 is operatively coupled to a display 622, which presentsimages such as windows to the user on a graphical user interface 618B.The computer 602 may be coupled to other devices, such as a keyboard614, a mouse device 616, a printer, etc. Of course, those skilled in theart will recognize that any combination of the above components, or anynumber of different components, peripherals, and other devices, may beused with the computer 602.

Generally, the computer 602 operates under control of an operatingsystem 608 stored in the memory 606, and interfaces with the user toaccept inputs and commands and to present results through a graphicaluser interface (GUI) module 618A. Although the GUI module 618B isdepicted as a separate module, the instructions performing the GUIfunctions can be resident or distributed in the operating system 608,the computer program 610, or implemented with special purpose memory andprocessors. The computer 602 also implements a compiler 612 which allowsan application program 610 written in a programming language such asJava, C++, C#, or other language to be translated into processor 604readable code. After completion, the application 610 accesses andmanipulates data stored in the memory 606 of the computer 602 using therelationships and logic that was generated using the compiler 612.Analogous results can be accomplished with field programmable gatearrays (FPGAs). The computer 602 also optionally comprises an externalcommunication device such as a modem, satellite link, Ethernet card, orother device for communicating with other computers.

In one embodiment, instructions implementing the operating system 608,the computer program 610, and the compiler 612 are tangibly embodied ina computer-readable medium, e.g., data storage device 620, which couldinclude one or more fixed or removable data storage devices, such as azip drive, floppy disc drive 624, hard drive, CD-ROM drive, tape drive,etc. Further, the operating system 608 and the computer program 610 arecomprised of instructions which, when read and executed by the computer602, causes the computer 602 to perform the operations herein described.Computer program 610 and/or operating instructions may also be tangiblyembodied in memory 606 and/or data communications devices 630, therebymaking a computer program product or article of manufacture. As such,the terms “article of manufacture,” “program storage device” and“computer program product” as used herein are intended to encompass acomputer program accessible from any computer readable device or media.

It is understood that the foregoing embodiment of the computer systemincludes peripherals (e.g. display 622, GUI module 618A, GUI 618, mousedevice 616, keyboard 614, printer 628 or compiler 612) that may beuseful in the ground station 118 and similar applications, butunnecessary not included in the other processing elements.

Those skilled in the art will recognize many modifications may be madeto this configuration without departing from the scope of the presentdisclosure. For example, those skilled in the art will recognize thatany combination of the above components, or any number of differentcomponents, peripherals, and other devices, may be used.

Conclusion

This concludes the description of the preferred embodiments of thepresent disclosure. The foregoing description of the preferredembodiment has been presented for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit thedisclosure to the precise form disclosed. Many modifications andvariations are possible in light of the above teaching. It is intendedthat the scope of rights be limited not by this detailed description,but rather by the claims appended hereto.

What is claimed is:
 1. A method of generating an image, comprising:reading a plurality of raw sensor data sets, each raw sensor data setread from one or more sensors at a plurality of sensor inertial states;generating an estimate of each of the plurality of sensor inertialstates; retaining each of the raw sensor data sets while performingsteps comprising: generating an image, the image generated at least inpart from the plurality of estimated sensor inertial states and theplurality of raw sensor data sets; and generating an updated estimate ofat least one of the sensor inertial states, the updated estimate of theat least one of the sensor inertial states generated at least in partfrom the generated image and the plurality of estimated sensor inertialstates; generating an enhanced image from the retained raw sensor datasets and the updated estimate of the at least one of the sensor inertialstates.
 2. The method of claim 1, wherein: generating an updatedestimate of at least one of the sensor inertial states, the updatedestimate of the at least one of the sensor inertial states generated atleast in part from the generated image and the plurality of estimatedsensor inertial states comprises: generating an updated estimate of eachof the plurality of sensor inertial states, the updated estimate of eachof the plurality of sensor inertial states generated at least in partfrom the generated image and the plurality of estimated sensor inertialstates.
 3. The method of claim 2, wherein: reading a plurality of rawsensor data sets, each raw sensor data set read from one or more sensorsat a plurality of sensor inertial states comprises: reading a pluralityof raw sensor data sets, each raw sensor data set read from a singlesensor at the plurality of sensor inertial states.
 4. The method ofclaim 3, wherein: the estimate of each of the plurality of sensorinertial states comprises an estimate of each of a plurality of sensorlocations associated with the plurality of raw sensor data sets; andgenerating an updated estimate of each of the plurality of sensorinertial states, the updated estimate of each of the plurality of sensorinertial states generated at least in part from the generated image andthe plurality of estimated sensor inertial states comprises: generatingan updated estimate of each of the plurality of sensor locations, theupdated estimate of each of the plurality of sensor locations generatedat least in part from the generated image and the plurality of estimatedsensor locations.
 5. The method of claim 4, wherein generating anupdated estimate of each of the plurality of sensor locations, theupdated estimate of each of the plurality of sensor locations generatedat least in part from the generated image and the plurality of estimatedsensor locations comprises: generating a georegistered image from thegenerated image and a reference image database; and generating theupdated estimate of each of the plurality of sensor locations at leastin part from the georegistered image.
 6. The method of claim 5, furthercomprising: generating an enhanced georegistered image from the enhancedimage and the updated estimate of each of the plurality of sensorlocations.
 7. The method of claim 6, wherein the sensor comprises asynthetic aperture radar (SAR).
 8. An apparatus of generating an image,comprising: a sensor for generating raw data; an image processor,communicatively coupled to the sensor, for reading a plurality of rawsensor data sets from the sensor, each raw sensor data set read from thesensor at a plurality of sensor inertial states; an inertial navigationsystem, communicatively coupled to the sensor, for generating anestimate of each of the plurality of sensor inertial states; an imageprocessor, communicatively coupled to the sensor and the inertialnavigation system, for generating an image, the image generated at leastin part from the plurality of estimated sensor inertial states and theplurality of raw sensor data sets; and a georegistration system,communicatively coupled to the inertial navigation system and the imageprocessor, for generating an updated estimate of at least one of thesensor inertial states, the updated estimate of the at least one of thesensor inertial states generated at least in part from the generatedimage and the plurality of estimated sensor inertial states; wherein theimage processor generates an enhanced image from the retained raw sensordata sets and the updated estimate of the at least one of the sensorinertial states and the sensor retains each of the raw sensor data setswhile the image is generated from the raw sensor data and the updatedestimate of the at least one of the sensor inertial states is generated.9. The apparatus of claim 8, wherein: the georegistration systemgenerates an updated estimate of each of the plurality of sensorinertial states least in part from the generated image and the pluralityof estimated sensor inertial states.
 10. The apparatus of claim 9,wherein: the estimate of each of the plurality of sensor inertial statescomprises an estimate of a plurality of sensor locations associated withthe plurality of raw sensor data sets; and the georegistration systemgenerates an updated estimate of each of the plurality of sensorlocations, the updated estimate of each of the plurality of sensorlocations generated at least in part from the generated image and theplurality of estimated sensor locations.
 11. The apparatus of claim 10,wherein the georegistration system: generates the updated estimate ofeach of the plurality of sensor locations at least in part from thegenerated image and the plurality of estimated sensor locations bygenerating a georegistered image from the generated image and areference image database; and generates the updated estimate of each ofthe plurality of sensor locations at least in part from thegeoregistered image.
 12. The apparatus of claim 11, wherein: thegeoregistration system further generates an enhanced georegistered imagefrom the enhanced image and the updated estimate of each of theplurality of sensor locations.
 13. The apparatus of claim 12, whereinthe sensor comprises a synthetic aperture radar (SAR).
 14. An apparatusfor generating an image, comprising: means for reading a plurality ofraw sensor data sets, each raw sensor data set read from one or moresensors at a plurality of sensor inertial states; generating an estimateof each of the plurality of sensor inertial states; retaining each ofthe raw sensor data sets while performing steps comprising: generatingan image, the image generated at least in part from the plurality ofestimated sensor inertial states and the plurality of raw sensor datasets; and generating an updated estimate of at least one of the sensorinertial states, the updated estimate of the at least one of the sensorinertial states generated at least in part from the generated image andthe plurality of estimated sensor inertial states; generating anenhanced image from the retained raw sensor data sets and the updatedestimate of the at least one of the sensor inertial states.
 15. Theapparatus of claim 14, wherein: generating an updated estimate of atleast one of the sensor inertial states, the updated estimate of the atleast one of the sensor inertial states generated at least in part fromthe generated image and the plurality of estimated sensor inertialstates comprises: generating an updated estimate of each of theplurality of sensor inertial states, the updated estimate of each of theplurality of sensor inertial states generated at least in part from thegenerated image and the plurality of estimated sensor inertial states.16. The method of claim 15, wherein: reading a plurality of raw sensordata sets, each raw sensor data set read from one or more sensors at aplurality of sensor inertial states comprises: reading a plurality ofraw sensor data sets, each raw sensor data set read from a single sensorat the plurality of sensor inertial states.
 17. The apparatus of claim16, wherein: the estimate of each of the plurality of sensor inertialstates comprises an of a plurality of sensor locations associated withthe plurality of raw sensor data sets; and generating an updatedestimate of each of the plurality of sensor inertial states, the updatedestimate of each of the plurality of sensor inertial states generated atleast in part from the generated image and the plurality of estimatedsensor inertial states comprises: generating an updated estimate of eachof the plurality of sensor locations, the updated estimate of each ofthe plurality of sensor locations generated at least in part from thegenerated image and the plurality of estimated sensor locations.
 18. Theapparatus of claim 17, wherein generating an updated estimate of each ofthe plurality of sensor locations, the updated estimate of each of theplurality of sensor locations generated at least in part from thegenerated image and the plurality of estimated sensor locationscomprises: generating a georegistered image from the generated image anda reference image database; and generating the updated estimate of eachof the plurality of sensor locations at least in part from thegeoregistered image.
 19. The apparatus of claim 18, further comprising:generating an enhanced georegistered image from the enhanced image andthe updated estimate of each of the plurality of sensor locations. 20.The apparatus of claim 19, wherein the sensor comprises a syntheticaperture radar (SAR).